Autofocus and/or autoscaling in telesurgery

ABSTRACT

Robotic, telerobotic, and/or telesurgical devices, systems, and methods take advantage of robotic structures and data to calculate changes in the focus of an image capture device in response to movement of the image capture device, a robotic end effector, or the like. As the size of an image of an object shown in the display device varies with changes in a separation distance between that object and the image capture device used to capture the image, a scale factor between a movement command input may be changed in response to moving an input device or a corresponding master/slave robotic movement command of the system. This may enhance the perceived correlation between the input commands and the robotic movements as they appear in the image presented to the system operator.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of prior application Ser. No.11/239,661 filed Sep. 29, 2005, the full disclosure of which isincorporated by reference

BACKGROUND OF THE INVENTION

The present invention is generally related to telesurgical devices,systems, and methods. In an exemplary embodiment, the invention providessystems and methods for robotically altering a focus, optical scaling,and/or scaling factor of a robotic surgical system in response torobotic movements, preferably so as to maintain focus at a fixedlocation in space during movement of an image capture device, so as tomaintain focus on a moving robotic tool, or the like; and/or so as toadjust the scale of robotic end effector movements corresponding toinput commands in a master/slave telerobotic system so as to compensatefor the changes in scale of an object shown in a display, and the like.

Minimally invasive medical techniques are intended to reduce the amountof extraneous tissue which is damaged during diagnostic or surgicalprocedures, thereby reducing patient recovery time, discomfort, anddeleterious side effects. One effect of minimally invasive surgery, forexample, may be reduced post-operative hospital recovery times. Becausethe average hospital stay for a standard surgery is typicallysignificantly longer than the average stay for an analogous minimallyinvasive surgery, increased use of minimally invasive techniques couldsave millions of dollars in hospital costs each year. While many of thesurgeries performed each year in the United States could potentially beperformed in a minimally invasive manner, only a portion of the currentsurgeries use these advantageous techniques due to limitations inminimally invasive surgical instruments and the additional surgicaltraining involved in mastering them.

Minimally invasive robotic surgical or telesurgical systems have beendeveloped to increase a surgeon's dexterity and avoid some of thelimitations on traditional minimally invasive techniques. Intelesurgery, the surgeon uses some form of remote control, e.g., aservomechanism or the like, to manipulate surgical instrument movements.In telesurgery systems, the surgeon can be provided with an image of thesurgical site at the surgical workstation. While viewing a two or threedimensional image of the surgical site on a display, the surgeonperforms the surgical procedures on the patient by manipulating mastercontrol devices, which in turn control motion of the servomechanicallyoperated instruments.

The servomechanism used for telesurgery will often accept input from twomaster controllers (one for each of the surgeon's hands) and may includetwo or more robotic arms or manipulators, on each of which a surgicalinstrument is mounted. Operative communication between mastercontrollers and associated robotic arm and instrument assemblies istypically achieved through a control system. The control systemtypically includes at least one processor which relays input commandsfrom the master controllers to the associated robotic arm and instrumentassemblies, and back from the instrument and arm assemblies to theassociated master controllers (in the case of, e.g., force feedback orthe like). One example of a robotic surgical system is the DaVinci®system available from Intuitive Surgical, Inc. of Mountain View, Calif.

The new telesurgical devices have significantly advanced the art,providing huge potential improvements in endoscopic procedures. However,as with many such advances, still further improvements would bedesirable. In particular, it is generally beneficial to provide clearand precise displays of the surgical environment and treatments to asurgeon working with a telesurgical system. Three dimensional imagedisplays significantly enhance the surgeon's ability to interact withthe tissues and visually guide the procedure, as the visual input may bemore complete (as compared to open surgical procedures) than the tactilefeedback provided by some robotic systems. When placing a heightenedreliance on visual input, any loss of focus by the imaging system may beparticularly distracting. Additionally, while the known robotic surgicalsystems may provide good correlation between movement of the inputdevices and movement of the robotic instruments in many circumstances,the correlation might still benefit from further improvements.

In general, it would be desirable to provide improved telesurgicaland/or telerobotic devices, systems, and methods. It would be, forexample, advantageous to provide new approaches for maintaining clarityof the visual display presented to surgeons and other system operatorsof such telesurgical and telerobotic devices. It would also, forexample, be helpful to provide enhanced correlations between the inputmovements and the robotic end effector movements calculated by theprocessor of the system, particularly as the configuration of therobotic procedure undergoes changes as the procedure progresses.

BRIEF SUMMARY OF THE INVENTION

The present invention generally provides improved robotic, telerobotic,and/or telesurgical devices, systems, and methods. Exemplary embodimentstake advantage of the robotic structures and data of these systems,along with new and/or modified structural components, to calculatechanges in the focus of an image capture device in response to movementof the image capture device, a robotic end effector, or the like. As thesize of an image of an object shown in the display device varies (forexample, with changes in a separation distance between that object andthe image capture device used to capture the image), some embodimentsmay change the motion scale factor between a movement command input bymoving an input device and a corresponding master/slave robotic movementcommand of the system. This may enhance the perceived correlationbetween the input commands and the robotic movements as they appear inthe image presented to the system operator.

In a first aspect, the invention provides a surgical robotic systemcomprising a image capture device having a variable focus. A roboticlinkage movably extends from the base to the image capture device, andan actuator is coupled to the variable focus of the image capturedevice. A processor couples the robotic linkage to the actuator. Theprocessor transmits a command signal to the actuator in response to amovement of the linkage such that a change in the variable focuscompensates for movement of the image capture device.

In another aspect, the invention provides a surgical system comprisingan image capture device for capturing an image of an object. A displayis coupled to the image capture device so as to show the image. Adisplay scale of the object in the image varies with a separationdistance between the object and the image capture device. A roboticlinkage effects relative movement between the object and the imagecapture device. An input device is provided to allow a master/slaveinput command to be entered into the system. A processor couples therobotic linkage to the input device. The processor determines therelative movement corresponding to the movement command per a motionscale factor. The processor alters the motion scale factor in responseto the relative movement so as to compensate for changes in the displayscale.

In yet another aspect, the invention provides a surgical robotic method.The method comprises capturing an image of an object at a surgical sitewith an image capture device. The object or the image capture device ismoved robotically with a relative movement. A new robotic motion scalefactor or focus is determined in response to the relative movement.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic block diagram illustrating a telesurgical systemin which focus and/or a motion scale factor is adjusted in response torobotic movements.

FIG. 1A shows a three dimensional view of an operator station of thetelesurgical system of FIG. 1.

FIG. 2 shows a three-dimensional view of a patient-side cart or surgicalstation of the telesurgical system, the cart carrying three roboticallycontrolled arms, the movement of the arms being remotely controllablefrom the operator station shown in FIG. 1A.

FIG. 3 shows a side view of a robotic arm and surgical instrumentassembly.

FIG. 4 shows a three-dimensional view of a surgical instrument.

FIG. 5 shows, at an enlarged scale, a wrist member and end effector ofthe surgical instrument shown in FIG. 3, the wrist member and endeffector being movably mounted on a working end of a shaft of thesurgical instrument.

FIG. 6 shows a three-dimensional view of the master control deviceshowing the wrist gimbal mounted on the articulated arm portion.

FIG. 7 shows a schematic three-dimensional drawing indicating thepositions of the end effectors relative to a viewing end of an endoscopeand the corresponding positions of master control devices relative tothe eyes of an operator, typically a surgeon.

FIG. 8 shows a schematic three-dimensional drawing indicating theposition and orientation of an end effector relative to an imagingCartesian coordinate reference system.

FIG. 9 shows a schematic three-dimensional drawing indicating theposition and orientation of a pincer formation of the master controldevice relative to an eye Cartesian coordinate reference system.

FIG. 10 shows a schematic side view of part of the surgical station ofthe minimally invasive surgical apparatus indicating the location ofCartesian reference coordinate systems used to determine the positionand orientation of an end effector relative an image capturing device.

FIG. 11 shows a schematic side view of part of the operator station ofthe minimally invasive surgical apparatus indicating the location ofCartesian reference coordinate systems used by the control systems ofthe minimally invasive surgical apparatus to determine the position andorientation of the input device relative to an eye of the systemoperator.

FIG. 12 schematically illustrates a high level control architecturemodel of a master/slave surgical robotic system.

FIG. 13 shows a block diagram representing control steps followed by thecontrol system of the minimally invasive surgical apparatus in effectingcontrol between input device positional and orientational movement andend effector positional and orientational movement.

FIG. 14 shows a fragmentary portion of the insertion portion of anendoscope for use with the present invention.

FIG. 15 is a schematic representation of a correlation between a focussetting of an image capture device and a separation distance between theimage capture device and the focus point.

FIG. 16 is a flowchart schematically illustrating a method for adjustinga focus and/or a movement scaling of the telesurgical system of FIG. 1.

DETAILED DESCRIPTION OF THE INVENTION

The present invention generally provides improved robotic, telerobotic,and/or telesurgical devices, systems, and methods. Embodiments of theinvention may be particularly well suited for minimally invasive or openrobotic surgical procedures, often using a master/slave telesurgicalsystem. Although this invention will often be described below in thecontext of a robotic, computer-enhanced surgical system, the inventionmay also have applications beyond robotic surgical systems to anysurgical environment that utilizes a camera to provide an image of thesurgical site to the surgeon, as is often provided for minimallyinvasive surgical procedures performed using laparoscopic instrumentsand the like.

Embodiments of the invention will often comprise a surgical systemhaving a camera or other image capture device to provide an image of thesurgical site for viewing by the surgeon. By including an encoder orother sensor coupled to the focusing mechanism of the camera,information can be provided that is useful in many different ways.Hence, many embodiments of the invention will include a sensor formeasuring a state of the focus mechanism, the sensor typicallycomprising an encoder, a potentiometer, or the like coupled to the focusmechanism of the image capture device, such as to the camera head of anendoscopic camera system. For ease of reference, the followingdescription will often refer simply to an encoder as sensing theposition or state of the focusing mechanism, although a wide variety ofalternative state sensing systems might also be employed. Similarly, thesystem will often be described with reference to an image capture devicecomprising an endoscope, as well as to a camera head of the imagecapture device, which is typically operatively connected to opticalcomponents of the image capture device. These insertable image capturedevices will often include at least a portion which is suitable forpositioning within a patient so as to be able to provide an image of aninternal surgical site to a system operator. Advantageously, thedevices, systems, and methods described herein may optionally employsignificant portions of commercially available robotic surgical systems,including the DaVinci® surgical system available from IntuitiveSurgical, Inc. of Sunnyvale, Calif.

Referring now to FIG. 1, a telesurgical system 1 allows a systemoperator O to perform a surgical treatment on an internal tissue ofpatient P. An image capture device 2 preferably obtains threedimensional images of the internal tissue, and an image of the tissue isdisplayed to the system operator O on a display device 3. Whileobserving the image in the display device, and by reference to thatimage, the system operator manipulates one or more handles of an inputdevice 4.

In response to signals from the input device, the processor 5 calculatesmovement of a treatment probe or instrument 6 for manipulation oftissues. More specifically, processor 5 transmits signals correspondingto the calculated movements desired by probe 6 to an associated roboticmanipulator 7, and the manipulator in response effects movement of theprobe. Probe 6 will often be mounted to an instrument holder ofmanipulator 7, and may also include additional degrees of freedom.

Along with providing movement of treatment probe 6, telesurgical system1 also allows movement and adjustment to image capture device 2, hereusing another manipulator 7. The manipulator 7 supporting the imagecapture device may be repositioned by system operator O throughappropriate inputs into the input device 4. Processor 5 calculatesappropriate movements of the manipulator 7 supporting image capturedevice 2 in response to these inputs, and the manipulator provides thosemovements in response to signals from the processor. Image capturedevice 2 transmits image signals to display 3, and may also provideimage and/or other signals for use by processor 1, such as providingimage signals for image processing techniques or the like.

As illustrated in FIG. 1, positional information from the focus encoderor other sensor of an encoder/actuator 8 is provided to processor 1 toallow the processor to determine where the camera should focus so as toallow the surgeon to continue to operate without interruption. Theprocessor transmits focus control signals to encoder/actuator 8 usingthe information provided regarding the current focus state from theencoder/actuator, and also using information from manipulators 7regarding the position or state of the image capture device 2, probe 6,and the like. In some embodiments, processor 5 may transmit signals to afocus encoder/actuator 8 so as to maintain the focus of image capturedevice 2 at a given location in a coordinate frame of reference of theinternal surgical site of patient P, such as a point in Cartesiancoordinate reference frame Pxyz. In other embodiments, the signalstransmitted from processor 5 to focus encoder/actuator 8 may maintainthe focus of image capture device 2 on a structure or surface of tissuemanipulation instrument or probe 6. While the data path between focusencoder/actuator 8 and processor 1 is schematically shown as a separatepathway, communication between the focusing mechanism and the processorwill often be handled as a multiplexed or separated channel portion ofthe communication between the associated manipulator 7 and theprocessor.

Robotic autofocus may be implemented in a variety of differing methodsand systems. In many embodiments, the camera/endoscope combination willinitially be adjusted to focus at a particular point in the surgicalfield, such as a point in reference frame Pxyz. This initial focus pointcan be achieved in a number of different ways. For example, whileholding the camera/endoscope substantially stationary with respect tothe surgical site, the surgeon may manually input focus commands so thatthe camera focuses on a desired location. The desired location maycomprise the tip of a robotic tool, instrument, or probe at the surgicalsite, a portion of the surgical site itself, a coronary vessel during ananastomotic procedure, or a heart valve during a valve repair orreplacement procedure (for example). Such manual focus can be achievedthrough the use of a surgeon input device, such as a foot pedal/switch,a manual toggle, or a voice control system that commands the camera headfocusing mechanism to move until the desired point of focus is achieved.

Alternatively, the camera may automatically focus on an object withinthe surgical field utilizing any of a wide variety of commerciallyavailable autofocus technologies. Suitable systems may include activesystems using ultrasound or infrared signals, passive image-analysissystems, and/or the like, including those described in, for example,U.S. Pat. Nos. 4,983,033, 4,602,861, and/or 4,843,416, or that includedin the SX-70 rangefinder commercialized by Polaroid. The point of theinitial autofocus may coincide with a sharp edge of a substantiallystationary surgical instrument, or of a target located on one or more ofthe surgical instruments, a target attached to one or more structures ofthe internal surgical site, or the like. Such autofocus methods mayagain be achieved through an appropriate surgeon's input device, such asan initial focus button that would cause the camera/endoscope toautomatically focus on the tool tips, or the like. Alternatively, thesystem processor could include a detector to detect when an instrumenthaving an appropriate target was placed within the surgical field. Oncethe detector determines that a suitable target was present within thefield, the endoscope/camera could automatically focus without having tobe commanded to do so by the surgeon.

Regardless of the manner of achieving the point of initial focus(whether manual, automatic, or otherwise), this point may be referred toas the initial focus point. Upon capturing this initial focus point, theposition or state occupied by the camera's focusing mechanismcorresponds to this initial focus point can be known by processor 5 viathe state information provided by encoder/actuator 8. System 1 canmaintain that particular focus point regardless of subsequent movementby the surgeon of the image capture device 2 using a “Stay in Focus”function of processor 5. For example, the surgeon may move the imagecapture device 2 away from the surgical site to capture a wider field ofview. Alternatively, the surgeon may move the endoscope toward thesurgical site along the axis of the endoscope for a closer view.Regardless of the type of movement, the state of the focusing mechanismat the initial focus point, the particular optical parameters of theendoscope, and/or the relationship between the focus mechanism state andthe focus point distance separating the endoscope from the focus pointand the specific movements of the endoscope by manipulator 7 are allknown. From this information, focus encoder/actuator 8 may be driven byprocessor 5 in response to movement of manipulator 1 so as to maintainthe point of focus of image capture device 2 at a fixed point inreference frame Pxyz within patient P, with the fixed point being set asthe initial focus point.

Calculation of the separation distance between a robotically movingimage capture device and a particular point in space within patient Pmay be facilitated by tracking the motion of the manipulator 7supporting the image capture device. Similarly, when the focus target isa surface or structure of probe 6, monitoring motion of the manipulatorssupporting both the image capture device and probe will allowcalculation of changes in relative positioning between the image capturedevice and the point of focus. For example, as described in U.S. Pat.No. 6,424,885, the full disclosure of which is incorporated herein byreference, telesurgical control may be referenced into a Cartesiancoordinate system, which may be coupled to the image capture device soas to maintain coordination between master/slave input commands by thesystem operator and the movement of tissue manipulation instruments orprobe 6. The information regarding the robotic arm movements aregenerally known via various encoders or potentiometers of the roboticlinkage, and this information is often available to the processorcontrolling manipulator movements, including to the DaVinci™ surgicalsystem manufactured by Intuitive Surgical, Inc. The position informationfrom the manipulators 7 is fed to processor 5, which can then instructthe focus encoder/actuator 8 of the focus mechanism in image capturedevice 2. The instruction signals from processor 5 to the focusencoder/actuator may comprise, for example, a specific number of focusencoder counts to move in a desired direction to maintain the focus atan initial focus point, a desired change in focus potentiometer reading,or the like. Exemplary structures of the processor 5, input device 4,manipulators 7, and the like for performing these techniques will bedescribed with more detail with reference to the exemplary embodimentsof FIGS. 1A-14.

Referring to FIG. 1A of the drawings, an operator station or surgeon'sconsole of a minimally invasive telesurgical system is generallyindicated by reference numeral 200. The station 200 includes a viewer202 where an image of a surgical site is displayed in use. A support 204is provided on which an operator, typically a surgeon, can rest his orher forearms while gripping two master controls (not shown in FIG. 1A),one in each hand. The master controls are positioned in a space 206inwardly beyond the support 204. When using the control station 200, thesurgeon typically sits in a chair in front of the control station 200,positions his or her eyes in front of the viewer 202 and grips themaster controls one in each hand while resting his or her forearms onthe support 204.

In FIG. 2 of the drawings, a cart or surgical station of thetelesurgical system is generally indicated by reference numeral 300. Inuse, the cart 300 is positioned close to a patient requiring surgery andis then normally caused to remain stationary until a surgical procedureto be performed has been completed. The cart 300 typically has wheels orcastors to render it mobile. The station 200 is typically positionedremote from the cart 300 and can be separated from the cart 300 by agreat distance, even miles away, but will typically be used within anoperating room with the cart 300.

The cart 300 typically carries three robotic arm assemblies. One of therobotic arm assemblies, indicated by reference numeral 302, is arrangedto hold an image capturing device 304, e.g., a remote image device, anendoscope, or the like. Each of the two other arm assemblies 10, 10respectively, includes a surgical instrument 14. While described inportions of the following description with reference to endoscopicinstruments and/or image capture devices, many embodiments will insteadinclude intravascular and/or orthopedic instruments and remote imagingsystems.

The endoscope 304 has a viewing end 306 at a remote end of an elongateshaft thereof. It will be appreciated that the endoscope 304 has anelongate shaft to permit its viewing end 306 to be inserted through anentry port into an internal surgical site of a patient's body. Theendoscope 304 is operatively connected to the viewer 202 to display animage captured at its viewing end 306 on the viewer 202. Each roboticarm assembly 10, 10 is normally operatively connected to one of themaster controls. Thus, the movement of the robotic arm assemblies 10, 10is controlled by manipulation of the master controls. The instruments 14of the robotic arm assemblies 10, 10 have end effectors which aremounted on wrist members which are pivotally mounted on distal ends ofelongate shafts of the instruments 14, as is described in greater detailhereinbelow. It will be appreciated that the instrument 14 have elongateshafts to permit the end effectors to be inserted through entry portsinto the internal surgical site of a patient's body. Movement of the endeffectors relative to the ends of the shafts of the instruments 14 isalso, controlled by the master controls.

The robotic arms 10, 10, 302 are mounted on a carriage 97 by means ofsetup joint arms 95. The carriage 97 can be adjusted selectively to varyits height relative to a base 99 of the cart 300, as indicated by arrowsK. The setup joint arms 95 are arranged to enable the lateral positionsand orientations of the arms 10, 10, 302 to be varied relative to avertically extending column 93 of the cart 300. Accordingly, thepositions, orientations and heights of the arms 10, 10, 302 can beadjusted to facilitate passing the elongate shafts of the instruments 14and the endoscope 304 through the entry ports to desired positionsrelative to the surgical site. When the surgical instruments 14 andendoscope 304 are so positioned, the setup joint arms 95 and carriage 97are typically locked in position.

In FIG. 3 of the drawings, one of the robotic arm assemblies 10 is shownin greater detail. Each assembly 10 includes an articulated robotic arm12, and a surgical instrument, schematically and generally indicated byreference numeral 14, mounted thereon.

FIG. 4 indicates the general appearance of the surgical instrument 14 ingreater detail. The surgical instrument 14 includes an elongate shaft14.1. The wrist-like mechanism, generally indicated by reference numeral50, is located at a working end of the shaft 14.1. A housing 53,arranged releasably to couple the instrument to the robotic arm 12, islocated at an opposed end of the shaft 14.1. In FIG. 3, and when theinstrument 14 is coupled or mounted on the robotic arm 12, the shaft14.1 extends along an axis indicated at 14.2. The instrument 14 istypically releasably mounted on a carriage 11, which can be driven totranslate along a linear guide formation 24 of the arm 12 in thedirection of arrows P.

The robotic arm 12 is typically mounted on a base or platform at an endof its associated setup joint arm 95 by means of a bracket or mountingplate 16. The robotic arm 12 includes a cradle, generally indicated at18, an upper arm portion 20, a forearm portion 22 and the guideformation 24. The cradle 18 is pivotally mounted on the plate 16 in agimbaled fashion to permit rocking movement of the cradle 18 about apivot axis 28. The upper arm portion 20 includes link members 30, 32 andthe forearm portion 22 includes link members 34, 36. The link members30, 32 are pivotally mounted on the cradle 18 and are pivotallyconnected to the link members 34, 36. The link members 34, 36 arepivotally connected to the guide formation 24. The pivotal connectionsbetween the link members 30, 32, 34, 36, the cradle 18, and the guideformation 24 are arranged to constrain the robotic arm 12 to move in aspecific manner, specifically with a pivot center 49 is coincident withthe port of entry, such that movement of the arm does not excessivelyeffect the surrounding tissue at the port of entry.

Referring now to FIG. 5 of the drawings, the wrist-like mechanism 50will now be described in greater detail. In FIG. 5, the working end ofthe shaft 14.1 is indicated at 14.3. The wrist-like mechanism 50includes a wrist member 52. One end portion of the wrist member 52 ispivotally mounted in a clevis, generally indicated at 17, on the end14.3 of the shaft 14.1 by means of a pivotal connection 54. The wristmember 52 can pivot in the direction of arrows 56 about the pivotalconnection 54. An end effector, generally indicated by reference numeral58, is pivotally mounted on an opposed end of the wrist member 52. Theend effector 58 is in the form of, e.g., a clip applier for anchoringclips during a surgical procedure. Accordingly, the end effector 58 hastwo parts 58.1, 58.2 together defining a jaw-like arrangement.

It will be appreciated that the end effector can be in the form of anydesired surgical tool, e.g., having two members or fingers which pivotrelative to each other, such as scissors, pliers for use as needledrivers, or the like. Instead, it can include a single working member,e.g., a scalpel, cautery electrode, or the like. When a tool other thana clip applier is desired during the surgical procedure, the tool 14 issimply removed from its associated arm and replaced with an instrumentbearing the desired end effector, e.g., a scissors, or pliers, or thelike.

The end effector 58 is pivotally mounted in a clevis, generallyindicated by reference numeral 19, on an opposed end of the wrist member52, by means of a pivotal connection 60. It will be appreciated thatfree ends 11, 13 of the parts 58.1, 58.2 are angularly displaceableabout the pivotal connection 60 toward and away from each other asindicated by arrows 62, 63. It will further be appreciated that themembers 58.1, 58.2 can be displaced angularly about the pivotalconnection 60 to change the orientation of the end effector 58 as awhole, relative to the wrist member 52. Thus, each part 58.1, 58.2 isangularly displaceable about the pivotal connection 60 independently ofthe other, so that the end effector 58, as a whole, is angularlydisplaceable about the pivotal connection 60 as indicated in dashedlines in FIG. 5. Furthermore, the shaft 14.1 is rotatably mounted on thehousing 53 for rotation as indicated by the arrows 59. Thus, the endeffector 58 has three degrees of freedom of movement relative to the arm12, namely, rotation about the axis 14.2 as indicated by arrows 59,angular displacement as a whole about the pivot 60 and angulardisplacement about the pivot 54 as indicated by arrows 56. By moving theend effector within its three degrees of freedom of movement, itsorientation relative to the end 14.3 of the shaft 14.1 can selectivelybe varied. It will be appreciated that movement of the end effectorrelative to the end 14.3 of the shaft 14.1 is controlled byappropriately positioned actuators, e.g., electrical motors, or thelike, which respond to inputs from the associated master control todrive the end effector 58 to a desired orientation as dictated bymovement of the master control. Furthermore, appropriately positionedsensors, e.g., encoders, or potentiometers, or the like, are provided topermit the control system of the minimally invasive telesurgical systemto determine joint positions as described in greater detail hereinbelow.

One of the master controls 700, 700 is indicated in FIG. 6 of thedrawings. A hand held part or wrist gimbal of the master control device700 is generally indicated by reference numeral 699. Part 699 has anarticulated arm portion including a plurality of members or linksconnected together by pivotal connections or joints. The surgeon gripsthe part 699 by positioning his or her thumb and index finger over apincher formation. When the pincher formation is squeezed between thethumb and index finger, the fingers or end effector elements of the endeffector 58 close. When the thumb and index finger are moved apart thefingers of the end effector 58 move apart in sympathy with the movingapart of the pincher formation. The joints of the part 699 areoperatively connected to actuators, e.g., electric motors, or the like,to provide for, e.g., force feedback, gravity compensation, and/or thelike, as described in greater detail hereinbelow. Furthermore,appropriately positioned sensors, e.g., encoders, or potentiometers, orthe like, are positioned on each joint of the part 699, so as to enablejoint positions of the part 699 to be determined by the control system.

The part 699 is typically mounted on an articulated arm 712. Thearticulated arm 712 includes a plurality of links 714 connected togetherat pivotal connections or joints 714. It will be appreciated that alsothe articulated arm 712 has appropriately positioned actuators, e.g.,electric motors, or the like, to provide for, e.g., force feedback,gravity compensation, and/or the like. Furthermore, appropriatelypositioned sensors, e.g., encoders, or potentiometers, or the like, arepositioned on the joints so as to enable joint positions of thearticulated arm 712 to be determined by the control system.

To move the orientation of the end effector 58 and/or its position alonga translational path, the surgeon simply moves the pincher formation tocause the end effector 58 to move to where he wants the end effector 58to be in the image viewed in the viewer 202. Thus, the end effectorposition and/or orientation is caused to follow that of the pincherformation. The master control devices 700, 700 are typically mounted onthe station 200 through pivotal connections.

The electric motors and sensors associated with the robotic arms 12 andthe surgical instruments 14 mounted thereon, and the electric motors andsensors associated with the master control devices 700 are operativelylinked in the control system. The control system typically includes atleast one processor, typically a plurality of processors, for effectingcontrol between master control device input and responsive robotic armand surgical instrument output and for effecting control between roboticarm and surgical instrument input and responsive master control outputin the case of, e.g., force feedback.

In use, and as schematically indicated in FIG. 7 of the drawings, thesurgeon views the surgical site through the viewer 202. The end effector58 carried on each arm 12 is caused to perform positional andorientational movements in response to movement and action inputs on itsassociated master controls. The master controls are indicatedschematically at 700, 700. It will be appreciated that during a surgicalprocedure images of the end effectors 58 are captured by the endoscope304 together with the surgical site and are displayed on the viewer 202so that the surgeon sees the responsive movements and actions of the endeffectors 58 as he or she controls such movements and actions by meansof the master control devices 700, 700. The control system is arrangedto cause end effector orientational and positional movement as viewed inthe image at the viewer 202 to be mapped onto orientational andpositional movement of a pincher formation of the master control as willbe described in greater detail hereinbelow.

The operation of the control system of the minimally invasive surgicalapparatus will now be described in greater detail. In the descriptionwhich follows, the control system will be described with reference to asingle master control 700 and its associated robotic arm 12 and surgicalinstrument 14. The master control 700 will be referred to simply as“master” and its associated robotic arm 12 and surgical instrument 14will be referred to simply as “slave.”

Control between master and slave movement is achieved by comparingmaster position and orientation in an eye Cartesian coordinate referencesystem with slave position and orientation in a camera Cartesiancoordinate reference system. For ease of understanding and economy ofwords, the term “Cartesian coordinate reference system” will simply bereferred to as “frame” in the rest of this specification. Accordingly,when the master is stationary, the slave position and orientation withinthe camera frame is compared with the master position and orientation inthe eye frame, and should the position and/or orientation of the slavein the camera frame not correspond with the position and/or orientationof the master in the eye frame, the slave is caused to move to aposition and/or orientation in the camera frame at which its positionand/or orientation in the camera frame does correspond with the positionand/or orientation of the master in the eye frame. In FIG. 8, the cameraframe is generally indicated by reference numeral 610 and the eye frameis generally indicated by reference numeral 612 in FIG. 9.

When the master is moved into a new position and/or orientation in theeye frame 612, the new master position and/or orientation does notcorrespond with the previously corresponding slave position and/ororientation in the camera frame 610. The control system then causes theslave to move into a new position and/or orientation in the camera frame610 at which new position and/or orientation, its position andorientation in the camera frame 610 does correspond with the newposition and/or orientation of the master in the eye frame 612.

It will be appreciated that the control system includes at least one,and typically a plurality, of processors which compute new correspondingpositions and orientations of the slave in response to master movementinput commands on a continual basis determined by the processing cyclerate of the control system. A typical processing cycle rate of thecontrol system under discussion is about 1000 Hz or more, often beingabout 1300 Hz. Thus, when the master is moved from one position to anext position, the corresponding movement desired by the slave torespond is computed at about 1300 Hz. Naturally, the control system canhave any appropriate processing cycle rate depending on the processor orprocessors used in the control system. All real-time servocycleprocessing is preferably conducted on a DSP (Digital Signal Processor)chip. DSPs are preferable because of their constant calculationpredictability and reproducibility. A Share DSP from Analog Devices,Inc. of Massachusetts is an acceptable example of such a processor forperforming the functions described herein.

The camera frame 610 is positioned such that its origin 614 ispositioned at the viewing end 306 of the endoscope 304. Conveniently,the z axis of the camera frame 610 extends axially along a viewing axis616 of the endoscope 304. Although in FIG. 8, the viewing axis 616 isshown in coaxial alignment with a shaft axis of the endoscope 304, it isto be appreciated that the viewing axis 616 can be angled relativethereto. Thus, the endoscope can be in the form of an angled scope.Naturally, the x and y axes are positioned in a plane perpendicular tothe z axis. The endoscope is typically angularly displaceable about itsshaft axis. The x, y and z axes are fixed relative to the viewing axisof the endoscope 304 so as to displace angularly about the shaft axis insympathy with angular displacement of the endoscope 304 about its shaftaxis.

To enable the control system to determine slave position andorientation, a frame is defined on or attached to the end effector 58.This frame is referred to as an end effector frame or slave tip frame,in the rest of this specification, and is generally indicated byreference numeral 618. The end effector frame 618 has its origin at thepivotal connection 60. Conveniently, one of the axes e.g. the z axis, ofthe frame 618 is defined to extend along an axis of symmetry, or thelike, of the end effector 58. Naturally, the x and y axes then extendperpendicularly to the z axis. It will appreciated that the orientationof the slave is then defined by the orientation of the frame 618 havingits origin at the pivotal connection 60, relative to the camera frame610. Similarly, the position of the slave is then defined by theposition of the origin of the frame at 60 relative to the camera frame610.

Referring now to FIG. 9 of the drawings, the eye frame 612 is chosensuch that its origin corresponds with a position 201 where the surgeon'seyes are normally located when he or she is viewing the surgical site atthe viewer 202. The z axis extends along a line of sight of the surgeon,indicated by axis 620, when viewing the surgical site through the viewer202. Naturally, the x and y axes extend perpendicularly from the z axisat the origin 201. Conveniently, the y axis is chosen to extendgenerally vertically relative to the viewer 202 and the x axis is chosento extend generally horizontally relative to the viewer 202.

To enable the control system to determine master position andorientation within the viewer frame 612, a point on the master is chosenwhich defines an origin of a master or master tip frame, indicated byreference numeral 622. This point is chosen at a point of intersectionindicated by reference numeral 3A between axes of rotation 1 and 3 ofthe master. Conveniently, the z axis of the master frame 622 on themaster extends along an axis of symmetry of the pincher formation 706which extends coaxially along the rotational axis 1. The x and y axesthen extend perpendicularly from the axis of symmetry 1 at the origin3A. Accordingly, orientation of the master within the eye frame 612 isdefined by the orientation of the master frame 622 relative to the eyeframe 612. The position of the master in the eye frame 612 is defined bythe position of the origin 3A relative to the eye frame 612.

How the position and orientation of the slave within the camera frame610 is determined by the control system will now be described withreference to FIG. 10 of the drawings. FIG. 10 shows a schematic diagramof one of the robotic arm 12 and surgical instrument 14 assembliesmounted on the cart 300. When used for neurosurgery, cardiology, and/ororthopedic surgery, the linkages of the robotic arm and its associatedinstrument may be altered or tailored for positioning and moving aflexible catheter body, an orthopedic probe, or the like. However,before commencing with a description of FIG. 10, it is appropriate todescribe certain previously mentioned aspects of the surgical station300 which impact on the determination of the orientation and position ofthe slave relative to the camera frame 610.

In use, when it is desired to perform a surgical procedure by means ofthe minimally invasive surgical apparatus, the surgical station 300 ismoved into close proximity to a patient requiring the surgicalprocedure. The patient is normally supported on a surface such as anoperating table, or the like. To make allowance for support surfaces ofvarying height, and to make allowance for different positions of thesurgical station 300 relative to the surgical site at which the surgicalprocedure is to be performed, the surgical station 300 is provided withthe ability to have varying initial setup configurations. Accordingly,the robotic arms 12, 12, and the endoscope arm 302 are mounted on thecarriage 97 which is height-wise adjustable, as indicated by arrows K,relative to the base 99 of the cart 300, as can best be seen in FIGS. 2and 10 of the drawings. Furthermore, the robotic arms 12, 12 and theendoscope arm 302 are mounted on the carriage 97 by means of the setupjoint arms 95. Thus, the lateral position and orientation of the arms12, 12, 302 can be selected by moving the setup joint arms 95. Thus, atthe commencement of the surgical procedure, the cart 300 is moved intothe position in close proximity to the patient, an appropriate height ofthe carriage 97 is selected by moving it to an appropriate heightrelative to the base 99 and the surgical instruments 14 are movedrelative to the carriage 97 so as to introduce the shafts of theinstruments 14 and the endoscope 304 through the ports of entry and intopositions in which the end effectors 58 and the viewing end 306 of theendoscope 304 are appropriately positioned at the surgical site and thefulcrums are coincident with the ports of entry. Once the height andpositions are selected, the carriage 97 is locked at its appropriateheight and the setup joint arms 95 are locked in their positions andorientations. Normally, throughout the surgical procedure, the carriage97 is maintained at the selected height and similarly the setup jointarms 95 are maintained in their selected positions. However, if desired,either the endoscope or one or both of the instruments can be introducedthrough other ports of entry during the surgical procedure.

Returning now to FIG. 10, the determination by the control system of theposition and orientation of the slave within the camera frame 610 willnow be described. It will be appreciated that this is achieved by meansof one or more processors having a specific processing cycle rate. Thus,where appropriate, whenever position and orientation are referred to inthis specification, it should be borne in mind that a correspondingvelocity is also readily determined. The control system determines theposition and orientation of the slave within the camera frame 610 bydetermining the position and orientation of the slave relative to a cartframe 624 and by determining the orientation and position of theendoscope 304 with reference to the same cart frame 624. The cart frame624 has an origin indicated by reference numeral 626 in FIG. 10.

To determine the position and orientation of the slave relative to thecart frame 624, the position of a fulcrum frame 630 having its origin atthe fulcrum 49 is determined within the cart frame 624 as indicated bythe arrow 628 in dashed lines. It will be appreciated that the positionof the fulcrum 49 normally remains at the same location, coincident witha port of entry into the surgical site, throughout the surgicalprocedure. The position of the end effector frame 618 on the slave,having its origin at the pivotal connection 60, is then determinedrelative to the fulcrum frame 630 and the orientation of the endeffector frame 618 on the slave is also determined relative to thefulcrum frame 630. The position and orientation of the end effectorframe 618 relative to the cart frame is then determined by means ofroutine calculation using trigonometric relationships.

It will be appreciated that the robotic arm 302 of the endoscope 304 isconstrained to move in similar fashion to the robotic arm 10. Thus, theendoscope 304 when positioned with its viewing end 306 directed at thesurgical site, also defines a fulcrum coincident with its associatedport of entry into the surgical site. The endoscope arm 302 can bedriven to cause the endoscope 304 to move into a different positionduring a surgical procedure, to enable the surgeon to view the surgicalsite from a different position in the course of performing the surgicalprocedure. It will be appreciated that movement of the viewing end 306of the endoscope 304 is performed by varying the orientation of theendoscope 304 relative to its pivot center or fulcrum. The position andorientation of the camera frame 610 within the cart frame 624 isdetermined in similar fashion to the position and orientation of theslave within the cart frame 624. When the position and orientation ofthe camera frame 610 relative to the cart frame 624, and the positionand orientation of the slave relative to the cart frame 624 have beendetermined in this manner, the position and the orientation of the slaverelative to the camera frame 610 is readily determinable through routinecalculation using trigonometric relationships.

How the position and orientation of the master within the viewer frame612 is determined by the control system will now be described withreference to FIG. 11 of the drawings. FIG. 11 shows a schematic diagramof one of the master controls 700 at the operator station 200.

The operator station 200 optionally also includes setup joint arms, asindicated at 632, to enable the general location of the masters 700, 700to be varied to suit the surgeon. Thus, the general position of themasters 700, 700 can be selectively varied to bring the masters 700, 700into a general position at which they are comfortably positioned for thesurgeon. When the masters 700, 700 are thus comfortably positioned, thesetup joint arms 632 are locked in position and are normally maintainedin that position throughout the surgical procedure.

To determine the position and orientation of the master 700, asindicated in FIG. 11, within the eye frame 612, the position andorientation of the eye frame 612 relative to a surgeon's station frame634, and the position and orientation of the master 700 relative to thesurgeon's frame 634 is determined. The surgeon's station frame 634 hasits origin at a location which is normally stationary during thesurgical procedure, and is indicated at 636.

To determine the position and orientation of the master 700 relative tothe station frame 634, a position of a master setup frame 640 at an endof the setup joint arms 632 on which the master 700 is mounted, relativeto the station frame 636, is determined, as indicated by the arrow 638in dashed lines. The position and orientation of the master frame 622 onthe master 700 having its origin at 3A is then determined relative tothe master setup frame 640. In this manner, the position and orientationof the master frame 622 relative to the frame 634 can be determined bymeans of routine calculation using trigonometric relationships. Theposition and orientation of the eye frame 612 relative to the stationframe 634 is determined in similar fashion. It will be appreciated thatthe position of the viewer 202 relative to the rest of the surgeon'sconsole 200 can selectively be varied to suit the surgeon. The positionand orientation of the master frame 622 relative to the eye frame 612can then be determined from the position and orientation of the masterframe 622 and the eye frame 612 relative to the surgeon station frame634 by means of routine calculation using trigonometric relationships.

In the manner described above, the control system of the minimallyinvasive surgical apparatus determines the position and orientation ofthe end effector 58 by means of the end effector frame 618 in the cameraframe 610, and, likewise, determines the position and orientation of themaster by means of the master frame 622 relative to the eye frame 612.

As mentioned, the surgeon grips the master by locating his or her thumband index finger over the pincher formation 706. When the surgeon'sthumb and index finger are located on the pincher formation, the pointof intersection 3A is positioned inwardly of the thumb and index fingertips. The master frame having its origin at 3A is effectively mappedonto the end effector frame 618, having its origin at the pivotalconnection 60 of the end effector 58 as viewed by the surgeon in theviewer 202. Thus, when performing the surgical procedure, and thesurgeon manipulates the position and orientation of the pincherformation 706 to cause the position and orientation of the end effector58 to follow, it appears to the surgeon that his or her thumb and indexfinger are mapped onto the fingers of the end effector 58 and that thepivotal connection 60 of the end effector 58 corresponds with a virtualpivot point of the surgeon's thumb and index finger inwardly from thetips of the thumb and index finger.

Accordingly, as the surgical procedure is being performed the positionand orientation of the fingers of the end effector tracks orientationand position changes of the surgeon's thumb and index finger in anatural intuitive or superimposed fashion. Furthermore, actuation of theend effector 58, namely causing the end effector fingers selectively toopen and close, corresponds intuitively to the opening and closing ofthe surgeon's thumb and index finger. Thus, actuation of the endeffector 58 as viewed in the viewer 302 is performed by the surgeon in anatural intuitive manner, since the pivot point 60 of the end effector58 is appropriately mapped onto a virtual pivot point between thesurgeon's thumb and index finger.

Referring again to FIG. 10 of the drawings, the cart frame is chosen at624. It will be appreciated that determining the position of the fulcrumframe 630 relative to the cart frame 624 is achieved throughappropriately positioned sensors, such as potentiometers, encoders, orthe like. Conveniently, the fulcrum frame position 630 relative to thecart frame 624 is determined through two intermediate frames. One of theframes is a carriage guide frame 644 which has its origin at aconvenient location on a guide along which the carriage 97 is guided.The other frame, an arm platform frame indicated at 646 is positioned atan end of the setup joint arm 95 on which the robotic arm 12 is mounted.Thus, when slave position and orientation is determined relative to thecart frame 624, the carriage guide frame 644 position relative to thecart frame 624 is determined, then the platform frame 646 positionrelative to the carriage guide frame 644, then the fulcrum frame 630relative to the platform frame 646, and then the slave orientation andposition relative to the fulcrum frame 630, thereby to determine theslave position and orientation relative to the cart frame 624. It willbe appreciated that the slave position and orientation relative to thecart frame 624 is determined in this manner for each arm 10 and insimilar fashion for the camera frame 610, through its arm 302, relativeto the cart frame 624.

Referring to FIG. 11, the position and orientation of the master controlis determined by determining the position of a base frame 648 relativeto the surgeon's station frame 634, then determining the position of theplatform frame 640 relative to the base frame 648, and then determiningmaster position and orientation relative to the platform frame 640. Theposition and orientation of the master frame 622 relative to thesurgeon's station frame 634 is then readily determined through routinecalculation using trigonometric relationships. It will be appreciatedthat the position and orientation of the other master frame relative tothe surgeon console frame 634 is determined in a similar fashion.

FIG. 12 schematically illustrates a high level control architecture fora master/slave robotic system 1000. Beginning at the operator input, asurgeon 1002 moves an input device of a master manipulator 1004 byapplying manual or human forces f_(h) against the input device. Encodersof master manipulator 1004 generate master encoder signals e_(m) whichare interpreted by a master input/output processor 1006 to determine themaster joint positions θ_(m) The master joint positions are used togenerate Cartesian positions of the input device of the master x_(m)using a master kinematics model 1008.

Starting now with the input from the surgical environment 1018, thetissue structures in the surgical workspace will impose forces f_(e)against a surgical end effector (and possibly against other elements ofthe tool and/or manipulator). Environmental forces f_(e) from thesurgical environment 1018 alter position of the slave 1016, therebyaltering slave encoder values e_(s) transmitted to the slaveinput/output processor 1014. Slave input/output processor 1014interprets the slave encoder values to determine joint positions θ_(s),which are then used to generate Cartesian slave position signals x_(s)according to the slave kinematics processing block 1012.

The master and slave Cartesian positions x_(m), x_(s) are input intobilateral controller 1010, which uses these inputs to generate thedesired Cartesian forces to be applied by the slave f_(s) so that thesurgeon can manipulate the salve as desired to perform a surgicalprocedure. Additionally, bilateral controller 1010 uses the Cartesianmaster and slave positions x_(m), x_(s) to generate the desiredCartesian forces to be applied by the master f_(m) so as to provideforce feedback to the surgeon.

In general, bilateral controller 1010 will generate the slave and masterforces f_(s), f_(m) by mapping the Cartesian position of the master inthe master controller workspace with the Cartesian position of the endeffector in the surgical workspace according to a transformation.Preferably, the control system 1000 will derive the transformation inresponse to state variable signals provided from the imaging system onthat an image of the end effector in a display appears substantiallyconnected to the input device. These state variables will generallyindicate the Cartesian position of the field of view from the imagecapture device, as supplied by the slave manipulators supporting theimage capture device. Hence, coupling of the image capture manipulatorand slave end effector manipulator is beneficial for deriving thistransformation. Clearly, bilateral controller 1010 may be used tocontrol more than one slave arm, and/or may be provided with additionalinputs.

Based generally on the difference in position between the master and theslave in the mapped workspace, bilateral controller 1010 generatesCartesian slave force f_(s) to urge the slave to follow the position ofthe master. The slave kinematics 1012 are used to interpret theCartesian slave forces f_(s), to generate joint torques of the slaveτ_(s), which will result in the desired forces at the end effector.Slave input/output processor 1014 uses these joint torques to calculateslave motor currents i_(s), which reposition the slave x_(e) within thesurgical worksite.

The desired feedback forces from bilateral controller are similarlyinterpreted from Cartesian force on the master f_(m) based on the masterkinematics 1008 to generate master joint torques τ_(s). The master jointtorques are interpreted by the master input/output controller 1006 toprovide master motor current i_(m) to the master manipulator 1004, whichchanges the position of the hand held input device x_(h) in thesurgeon's hand.

It will be recognized that the control system 1000 illustrated in FIG.12 is a simplification. For example, the surgeon does not only applyforces against the master input device, but also moves the handle withinthe master workspace. Similarly, the motor current supplied to themotors of the master manipulator may not result in movement if thesurgeon maintains the position of the master controller. Nonetheless,the motor currents do result in tactile force feedback to the surgeonbased on the forces applied to the slave by the surgical environment.Additionally, while Cartesian coordinate mapping is preferred, the useof spherical, cylindrical, or other reference frames may provide atleast some of the advantages of the invention.

The control system, generally indicated by reference numeral 810, willnow be described in greater detail with reference to FIG. 13 of thedrawings, in which like reference numerals are used to designate similarparts or aspects, unless otherwise stated.

As mentioned earlier, the master control 700 has sensors, e.g.,encoders, or potentiometers, or the like, associated therewith to enablethe control system 810 to determine the position of the master control700 in joint space as it is moved from one position to a next positionon a continual basis during the course of performing a surgicalprocedure. In FIG. 13, signals from these positional sensors areindicated by arrow 814. Positional readings measured by the sensors at687 are read by the processor. It will be appreciated that since themaster control 700 includes a plurality of joints connecting one armmember thereof to the next, sufficient positional sensors are providedon the master 700 to enable the angular position of each arm memberrelative to the arm member to which it is joined to be determinedthereby to enable the position and orientation of the master frame 622on the master to be determined. As the angular positions of one armmember relative to the arm member to which it is joined is readcyclically by the processor 689 in response to movements induced on themaster control 700 by the surgeon, the angular positions arecontinuously changing. The processor at 689 reads these angularpositions and computes the rate at which these angular positions arechanging. Thus, the processor 689 reads angular positions and computesthe rate of angular change, or joint velocity, on a continual basiscorresponding to the system processing cycle time, i.e., 1300 Hz. Jointposition and joint velocity commands thus computed at 689 are then inputto the Forward Kinematics (FKIN) controller at 691, as already describedhereinabove.

At the FKIN controller 691, the positions and velocities in joint spaceare transformed into corresponding positions and velocities in Cartesianspace, relative to the eye frame 612. The FKIN controller 691 is aprocessor which typically employs a Jacobian (J) matrix to accomplishthis. It will be appreciated that the Jacobian matrix transforms angularpositions and velocities into corresponding positions and velocities inCartesian space by means of conventional trigonometric relationships.Thus, corresponding positions and velocities in Cartesian space, orCartesian velocity and position commands, are computed by the FKINcontroller 691 which correspond to Cartesian position and velocitychanges of the master frame 622 in the eye frame 612.

The velocity and the position in Cartesian space is input into aCartesian controller, indicated at 820, and into a motion scale andoffset converter, indicated at 822.

The minimally invasive surgical apparatus provides for a motion scalechange between master control input movement and responsive slave outputmovement. Thus, a motion scale can be selected where, for example, a1-inch movement of the master control 700 is transformed into acorresponding responsive ⅕-inch movement on the slave. At the motionscale and offset step 822, the Cartesian position and velocity valuesare scaled in accordance with the scale selected to perform the surgicalprocedure. Naturally, if a motion scale of 1:1 has been selected, nochange in motion scale is effected at 822. Similarly, offsets are takeninto account which determine the corresponding position and/ororientation of the end effector frame 618 in the camera frame 610relative to the position and orientation of the master frame 622 in theeye frame 612.

After a motion scale and offset step is performed at 822, a resultantdesired slave position and desired slave velocity in Cartesian space isinput to a simulated or virtual domain at 812, as indicated by arrows811. It will be appreciated that the labeling of the block 812 as asimulated or virtual domain is for identification only. Accordingly, thesimulated control described hereinbelow is performed by elements outsidethe block 812 also.

The simulated domain 812 will be described in greater detailhereinbelow. However, the steps imposed on the desired slave velocityand position in the virtual domain 812 will now be described broadly forease of understanding of the description which follows. A current slaveposition and velocity is continually monitored in the virtual orsimulated domain 812. The desired slave position and velocity iscompared with the current slave position and velocity. Should thedesired slave position and/or velocity as input from 822 not causetransgression of limitations, e.g., velocity and/or position and/orsingularity, and/or the like, as set in the virtual domain 812, asimilar Cartesian slave velocity and position is output from the virtualdomain 812 and input into an inverse scale and offset converter asindicated at 826. The similar velocity and position output in Cartesianspace from the virtual domain 812 is indicated by arrows 813 andcorresponds with actual commands in joint space output from the virtualdomain 812 as indicated by arrows 815 as will be described in greaterdetail hereinbelow. From the inverse scale and offset converter 826,which performs the scale and offset step of 822 in reverse, the revertedCartesian position and velocity is input into the Cartesian controllerat 820. At the Cartesian controller 820, the original Cartesian positionand velocities as output from the FKIN controller 691 is compared withthe Cartesian position and velocity input from the simulated domain 812.If no limitations were transgressed in the simulated domain 812 thevelocity and position values input from the FKIN controller 691 would bethe same as the velocity and position values input from the simulateddomain 812. In such a case, a zero error signal is generated by theCartesian controller 820.

In the event that the desired Cartesian slave position and velocityinput at 811 would transgress one or more set limitations, the desiredvalues are restricted to stay within the bounds of the limitations.Consequently, the Cartesian velocity and position forwarded from thesimulated domain 812 to the Cartesian controller 820 would then not bethe same as the values from the FKIN controller 691. In such a case,when the values are compared by the Cartesian controller 820, an errorsignal is generated.

Assuming that a zero error is generated at the Cartesian controller 820no signal is passed from the Cartesian controller or converter 820. Inthe case that an error signal is generated the signal is passed througha summation junction 827 to a master transpose kinematics controller828.

The error signal is typically used to calculate a Cartesian force. TheCartesian force is typically calculated, by way of example, inaccordance with the following formula:F _(CART) =K(Δx)+B(Δx)

where K is a spring constant, B is a damping constant, Δx is thedifference between the Cartesian velocity inputs to the Cartesiancontroller 820 and Δx is the difference between the Cartesian positioninputs to the Cartesian controller 820. It will be appreciated that foran orientational error, a corresponding torque in Cartesian space isdetermined in accordance with conventional methods.

The Cartesian force corresponds to an amount by which the desired slaveposition and/or velocity extends beyond the limitations imposed in thesimulated domain 812. The Cartesian force, which could result from avelocity limitation, a positional limitation, and/or a singularitylimitation, as described in greater detail below, is then converted intoa corresponding torque signal by means of the master transposekinematics controller 828 which typically includes a processor employinga Jacobian Transpose (J^(T)) matrix and kinematic relationships toconvert the Cartesian force to a corresponding torque in joint space.The torque thus determined is then input to a processor at 830 wherebyappropriate electrical currents to the motors associated with the master700 are computed and supplied to the motors. These torques are thenapplied on the motors operatively associated with the master control700. The effect of this is that the surgeon experiences a resistance onthe master control to either move it at the rate at which he or she isurging the master control to move, or to move it into the position intowhich he or she is urging the master control to move. The resistance tomovement on the master control is due to the torque on the motorsoperatively associated therewith. Accordingly, the higher the forceapplied on the master control to urge the master control to move to aposition beyond the imposed limitation, the higher the magnitude of theerror signal and the higher an opposing torque on the motors resistingdisplacement of the master control in the direction of that force.Similarly, the higher the velocity imposed on the master beyond thevelocity limitation, the higher the error signal and the higher theopposing torque on the motors associated with the master.

Referring once again to FIG. 13 of the drawings, the slave jointvelocity and position signal is passed from the simulated domain 812 toa joint controller 848. At the joint controller 848, the resultant jointvelocity and position signal is compared with the current joint positionand velocity. The current joint position and velocity is derived throughthe sensors on the slave as indicated at 849 after having been processedat an input processor 851 to yield slave current position and velocityin joint space.

The joint controller 848 computes the torques desired on the slavemotors to cause the slave to follow the resultant joint position andvelocity signal taking its current joint position and velocity intoaccount. The joint torques so determined are then routed to a feedbackprocessor at 852 and to an output processor at 854.

The joint torques are typically computed, by way of example, by means ofthe following formula:T=K(Δθ)+B(Δθ)

where K is a spring constant, B is a damping constant, Δθ is thedifference between the joint velocity inputs to the joint controller851, and Δθ is the difference between the joint position inputs to thejoint controller 851.

The output processor 854 determines the electrical currents to besupplied to the motors associated with the slave to yield the commandedtorques and causes the currents to be supplied to the motors asindicated by arrow 855.

From the feedback processor 852 force feedback is supplied to themaster. As mentioned earlier, force feedback is provided on the master700 whenever a limitation is induced in the simulated domain 812.Through the feedback processor 852 force feedback is provided directlyfrom the slave 798, in other words, not through a virtual or simulateddomain but through direct slave movement. This will be described ingreater detail hereinbelow.

As mentioned earlier, the slave indicated at 798 is provided with aplurality of sensors. These sensors are typically operatively connectedto pivotal joints on the robotic arm 10 and on the instrument 14.

These sensors are operatively linked to the processor at 851. It will beappreciated that these sensors determine current slave position. Shouldthe slave 798 be subjected to an external force great enough to inducereactive movement on the slave 798, the sensors will naturally detectsuch movement. Such an external force could originate from a variety ofsources such as when the robotic arm 10 is accidentally knocked, orknocks into the other robotic arm 10 or the endoscope arm 302, or thelike. As mentioned, the joint controller 848 computes torques desired tocause the slave 798 to follow the master 700. An external force on theslave 798 which causes its current position to vary also causes thedesired slave movement to follow the master to vary. Thus a compoundedjoint torque is generated by the joint controller 848, which torqueincludes the torque desired to move the slave to follow the master andthe torque desired to compensate for the reactive motion induced on theslave by the external force. The torque generated by the jointcontroller 848 is routed to the feedback processor at 852, as alreadymentioned. The feedback processor 852 analyzes the torque signal fromthe joint controller 848 and accentuates that part of the torque signalresulting from the extraneous force on the slave 798. The part of thetorque signal accentuated can be chosen depending on requirements. Inthis case, only the part of the torque signal relating to the roboticarm 12, 12, 302 joints are accentuated. The torque signal, after havingbeen processed in this way is routed to a kinematic mapping block 860from which a corresponding Cartesian force is determined. At thekinematic block 860, the information determining slave fulcrum positionrelative to the camera frame is input from 647 as indicated. Thus, theCartesian force is readily determined relative to the camera frame. ThisCartesian force is then passed through again step at 862 appropriatelyto vary the magnitude of the Cartesian force. The resultant force inCartesian space is then passed to the summation junction at 827 and isthen communicated to the master control 700 as described earlier.

Reference numeral 866 generally indicates another direct force feedbackpath of the control system 810, whereby direct force feedback issupplied to the master control 700. The path 866 includes one or moresensors which are not necessarily operatively connected to slave joints.These sensors can typically be in the form of force or pressure sensorsappropriately positioned on the surgical instrument 14, typically on theend effector 58. Thus, should the end effector 58 contact an extraneousbody, such as body tissue at the surgical site, it generates acorresponding signal proportionate to the force of contact. This signalis processed by a processor at 868 to yield a corresponding torque. Thistorque is passed to a kinematic mapping block 864, together withinformation from 647 to yield a corresponding Cartesian force relativeto the camera frame. From 864, the resultant force is passed through again block at 870 and then forwarded to the summation junction 827.Feedback is imparted on the master control 700 by means of torquesupplied to the motors operatively associated with the master control700 as described earlier. It will be appreciated that this can beachieved by means of any appropriate sensors such as current sensors,pressure sensors, accelerometers, proximity detecting sensors, or thelike. In some embodiments, resultant forces from kinematic mapping 864may be transmitted to an alternative presentation block 864.1 so as toindicate the applied forces in an alternative format to the surgeon.

Reference now is made to FIG. 14 wherein a distal end portion, or tip,260 of the insertion section of a flexible instrument or endoscope isshown. The insertion end of the instrument includes a pair of spacedviewing windows 262R and 262L and an illumination source 264 for viewingand illuminating a workspace to be observed. Light received at thewindows is focused by objective lens means, not shown, and transmittedthrough fiber-optic bundles to a pair of cameras at the operating end ofthe instrument, not shown. The camera outputs are converted to athree-dimensional image of the workspace which image is located adjacenthand-operated means at the operator's station, now shown. Right and leftsteerable catheters 268R and 268L pass through accessory channels in theendoscope body, which catheters are adapted for extension from thedistal end portion, as illustrated. End effectors 270R and 270L areprovided at the ends of the catheters which may comprise conventionalendoscopic instruments. Force sensors, not shown, also are insertedthrough the endoscope channels. Steerable catheters which includecontrol wires for controlling bending of the catheters and operation ofan end effector suitable for use with this invention are well know.Control motors for operation of the control wires are provided at theoperating end of the endoscope, which motors are included in aservomechanism of a type described above for operation of the steerablecatheters and associated end effectors from a remote operator's station.As with the other embodiments, the interfacing computer in theservomechanism system remaps the operator's hand motion into thecoordinate system of the end effectors, and images of the end effectorsare viewable adjacent the hand-operated controllers in a mannerdescribed above. Flexible catheter-based instruments and probes ofdifferent types may be employed in this embodiment of the invention.

In determining, establishing, and maintaining a desired focus point forthe endoscope or other image capture device, the controller or processorof the telesurgical system will generally take into account therelationship between the state of the focus mechanism and the distancebetween the endoscope tip and the focal point. Referring now to FIGS. 1and 15, there may be a variety of (typically non-linear) deterministiccompensation and correction curves for the camera/endoscope combinationthat relate particular focus settings to positions of an object in focusrelative to the endoscope. This information can be used to maintainfocus during movements of the object or endoscope. For example, for afirst non-linear compensation curve, if the endoscope moves two inchesradially away from the point of initial focus, moving from a separationdistance between the tip of the endoscope and the initial focus point oftwo inches to a separation distance of four inches, the focusingmechanism will generally move a first amount to compensate for this twoinch radial movement. However, if the endoscope moves so as to increasethe separation distance by two additional inches, from four inches ofseparation to six inches of separation between the endoscope tip and theinitial focus point, the focus setting may change by a second, differentamount on as to compensate for this additional two inch movement.Furthermore, the rate of change in focus setting and the absolute focussetting for a given distance may depend on other variable factors, suchas a magnification setting, or the like.

The focus setting/distance relationship graphically illustrated in FIG.15 may be developed in a variety of different ways. For example, thesystem may be tested at different magnification settings or the likethroughout a range of focus settings to identify the associateddistances, or the distances may be incrementally changed withappropriate focus settings being determined, measured, and recorded. Inaddition to parametric empirical studies, analysis of the optical trainof the image capture device using ray tracing or wavefront analyticaltechniques might also be employed. Still further alternatives may beavailable, including acquiring at least a portion of the data embodyingthe relationship of FIG. 15 from the supplier of one or more componentsof the image capture device.

As can be understood with reference to FIG. 8, purely lateral movementof the endoscope or focus point so as to provide the surgeon or systemoperator with a view of the surgical site that is from a differentangle, or the like, need not necessarily affect the focus setting of thecamera, particularly if the endoscope tip remains at a constant distancefrom the desired focus point at the surgical site. However, if amovement of the endoscope involves both lateral and longitudinal axialmovement of the endoscope, only the axial movement may be taken intoconsideration in adjusting the focus mechanism to maintain the focus atthe initial point of focus.

The “stay in focus” functionality described above, which may allow (forexample) telesurgical system 1 to maintain focus at a point in spacedespite movement of the image capture device, may be initiated inresponse to an input from the surgeon, such as the pressing of a buttonof input device 4. Activation of the button would inform the processor 5that the system operator O desires that the point of focus be maintainedat the location on which the endoscope and camera are focused at thetime the button is actuated. The input device may be embodied in avariety of different structures, including multiple buttons, areprogrammable button, a cursor or other object on a visual display suchas a graphic user interface, a voice control input, or a variety ofother input devices. Additionally, the “stay in focus” input device (orsome other input structure) could be used to designate one or more of anumber of different settings corresponding to different points of focus.Hence, after working at a site performing an anastomosis, for example,the surgeon might desire to pull the endoscope away from the site toprovide a wider view of the surgical site, and then to move to someother specific location within the surgical site that was outside theinitial field of view. Processor 5 may remember a plurality of positionsto be used as focus points for these differing views. The memory ofprocessor 5 may store two or more, three or more, or a large number ofpositions to be used as alternative focus points.

A variety of modifications of the system illustrated in FIG. 1 may alsobe provided. For example, in addition or instead of the focusencoder/actuator 8 coupled to image capture device 2, a magnificationencoder/actuator may be coupled to a variable magnification structure ofthe image capture device. The variable magnification structure maycomprise a selectable magnification optical train such as those usingmovable turrets having a plurality of alternative magnification lenses,zoom lens systems, and electronic pixel variation system (sometimesreferred to as an electronic zoom), or the like. Along with discretemagnification variation systems, continuous zoom systems may also beimplemented. Continuous zoom systems may be more easily implemented in asingle-channel endoscope than in a 2-channel or stereoscopic endoscopeand camera system, although maintaining relative optical magnificationof left and right eyes within a relatively tight correlation across thezoom range may allow the use of continuous zoom stereoscopic systems.

In a preferred embodiment, image capture device 2 has a firstmagnification setting and second magnification setting which differsfrom the first setting. An exemplary embodiment includes a dualmagnification camera head. Dual magnification camera heads may have anoptimum focus depth. At the optimum focus depth, switching from onemagnification to another magnification of a dual magnification camerahead does not affect the focus depth and does not require refocusing.However, switching magnifications when the camera is focused a pointthat differs significantly from the optimum focus depth may result inimage quality degradation until the focus is adjusted, the image capturedevice is moved to bring the surgical site back into focus, or the like.

In response to changes in magnification, as sensed by a magnificationencoder/actuator and as transmitted in signal form to processor 5, thefocus of image capture device 2 may be driven robotically using focusencoder/actuator 8 so as to maintain focus at the desired focus point.The change in focus may occur automatically without input (other thanthat used to alter magnification) by system operator O, and maycompensate for the switch in magnification using a relationship such asthat illustrated schematically in FIG. 15. This can be achieved bytaking advantage of sensors and/or actuators associated with the focusand/or magnification systems of image capture device 2, per thedirection of processor 5. By knowing where the camera is focused for thefirst magnification setting, the processor 5 can determine theappropriate focusing mechanism state for the second magnificationsetting so as to maintain the focus at the desired separation distancefrom the endoscope tip, the desired point in space, or the like. Thus,using information from focus and/or magnification sensors, processor 5can control the image capture device 2 to refocus or maintain focus ofthe focusing mechanism so that the desired focus point remains in focusdespite any magnification change.

A variety of still further alternative embodiments may also be provided.For example, a telesurgical system similar to system 1 of FIG. 1 maypermit focus to be maintained when one endoscope optical train that hasbeen coupled to a camera head is exchanged for another endoscope trainhaving differing optical and focus characteristics than the first. Theoptical characteristics of the endoscope structure may be programmedinto a memory of the endoscope structure and communicated to theprocessor of the telesurgical system when the endoscope structure ismounted or coupled to the system using techniques similar to thosedescribed in U.S. patent application Ser. No. 10/839,727, filed on May4, 2004, and assigned to the assignee of the subject application underthe title “Tool Memory-Based Software Upgrades for Robotic Surgery,” thefull disclosure of which is incorporated herein by reference.

Some embodiments of the autofocus system may rely on encoders or thelike which provide information on the relative position of the camera,focus mechanism, magnification mechanism, or the like. As the system mayonly maintain focus relative to an initial point of focus, the systemprocessor 5 need not necessarily know (for example) what the absolutestate or position of the camera focusing mechanism is during use. Insome embodiments, this may be acceptable during at least some modes ofoperation. In others, absolute position or state information regarding(for example) the magnification system, the focus system, or the likemay be addressed by having the mechanism run to a limit or calibratedstop, either manually or automatically (for example on power-up). Thismay allow the camera head focusing mechanism and/or processor 5 todetermine where the endoscope tip and/or focus point are relative toother objects in the surgical field of view. Other embodiments mayobtain absolute focusing mechanism state information using apotentiometer or the like. In some embodiments, the focus mechanism maybe driven using the absolute state information, for example, so as toinstruct the image capture device to focus at a particular location inspace.

While described primarily with reference to optical endoscopes having anelectronic camera head with a charge couple device (CCD) or the like, avariety of alternative image capture devices may also be employed. Forexample, the image capture device may make use of a remote imagingmodality such as ultrasound, X-ray or fluoroscopy, or the like, with oneexemplary remote imaging embodiment employing a Polaroid™ XS 70ultrasound system.

Information regarding the focus point of image capture device 2 may beused for a variety of purposes. For example, a surgeon or other systemoperator O will often have the camera focused on the portion of thesurgical site at which he or she is working at the time. Knowing thedepth at which the surgeon is working (for example, by identifying theseparation distance between an endoscope and the focus point) can beused to optimize or tailor other surgical systems to operate at theidentified depth or location. For example, U.S. Pat. No. 6,720,988entitled “A Stereo Imaging System and Method for Use in TeleroboticSystems”, the full disclosure of which is incorporated herein byreference, describes a method and apparatus for adjusting the workingdistance of a stereoscopic endoscope having two light channels. Workingdistance compensation is typically manually adjusted. By knowing thedepth at which the surgeon is working, however, and based on the focalpoint at which the endoscope is focused, the system processor may drivethe working distance compensation system to correspond to the focusdistance so as to provide a more coherent stereoscopic image to thesystem operator.

Other examples of subsystems which may be included in telesurgicalsystem 1 and which may benefit from information regarding the focusdistance include repositioning of the robotic arms and jointsresponsible for movement of the surgical tools (so that the arms arepositioned optimally for working at the desired depth), adjusting themotion scale of movement so that the operator's hand movements and inputinto input device 4 appear in proportion to the surgical site asdisplayed by display 3, irrigation at the surgical site at anappropriate distance from an integrated irrigation/endoscope structure,insufflation, alteration of tool wrist/elbow positioning (particularlyin systems having excess degrees of freedom), and the like. Additionalpotential uses of focus depth information include optimization ortailoring of illumination (so as to deposit an appropriate amount oflight), optimization of the camera sensor (for example, a CCD, a CMOS,or the like), and so on. Many of these structures may be implementedusing a tangible media (such as a magnetic disk, an optical disk, or thelike) embodying machine readable code with software instructions forperforming one, some, or all of the method steps described herein, oftenin combination with associated electronic, digital, and/or analog signalprocessing hardware.

In many embodiments, the devices, systems, and methods described hereinwill be useful for telepresence systems in which an image of a workingsite is displayed to a system operator at a position relative to masterinput controls manipulated by the operator in such a manner that theoperator substantially appears to be directly manipulating therobotically controlled end effectors. Such telepresence systems aredescribed, for example, in U.S. Pat. No. 5,808,665, the full disclosureof which is incorporated herein by reference, as well as in U.S. Pat.No. 6,424,885, which has previously been incorporated herein byreference. In these telepresence systems, surgeons and other systemoperators may benefit from having their hand movements appear todirectly correspond to the movements of the end effectors at the workingsite. A telepresence system may, for example, permit a surgeon to selectvarious scales of relative movement between the master input controlsand the slave manipulator end effector movements. For example, an inputfor selection between motion scales of 1:1, 2:1, and 5:1 may be providedfor a particular telepresence system, so that for every centimeter ofmovement of an end effector of the slave there are 1, 2, and 5centimeters of movement by the master input devices, respectively.Proper selection of motion scales may facilitate the performance of (forexample) very delicate surgical operations on small vasculature usingthe robotic tools, and may also help keep the surgical tool movementswith an appearance of being substantially connected to movement of themaster input devices.

While a plurality of alternatively selectable motion scales may servemany telepresence systems quite will, still further refinements arepossible. For example, the motion scale of movement between the endeffector and input device may benefit from changes when the camera headmagnification changes, when the endoscope zooms for a closer view of thesurgical site, or when the endoscope is simply brought closer to a toolor tissue of interest. If the movement scaling is not changed in thesecircumstances, the instruments may not appear as connected to thesurgeon's hand movements as might be ideal. Fortunately, this can beovercome by automatically adjusting the scale of movement appropriatelywith changes in the state of the image capture device, such as amagnification or optical scale of the endoscope and/or camera head, alocation of the endoscope tip relative to a worksite or tool, or thelike. This may help maintain the appearance that movements of thesurgeon's hands are more directly associated with movements of the toolsor treatment probes at the surgical site. Such association can beprovided when the system takes advantage of the depth of focus based onthe information provided from focus encoder/actuator 8 of image capturedevice 2, and relating that depth information to a correspondingmaster/slave proportional movement scaling. In some embodiments, thedepth information may be related to a predetermined set ofcorrespondence information (such as optical scale factors) between themaster and slave movements. The system may be pre-programmed with eithera mathematical relationship between different motion scales for a givendepth of field based on the geometric parameters of the system, or witha data lookup table. Regardless, the system may adjust the scale ofmovement for the desired relationship between focus depth and movementscaling, with this function optionally being referred to as autoscaling.

Autoscaling may be modified by having the endoscope remain focused onthe surgical instruments even when those instruments move. For example,the autofocus mechanism may focus on targets located on the instruments,as was previously described regarding the establishment of an initialpoint of focus. The autofocus function can also continually track motionof the end effectors during the surgical procedure. Without movement ofthe endoscope, the focus mechanism may be adjusted according to themovement of the end effectors (for example) of a tissue manipulationinstrument or probe 6, and thus according to the corresponding movementsof the targets on those probes. With such an arrangement, informationfrom the focus encoder/actuator 8 may be able to provide continuousinformation to processor 5 regarding where the surgical tool or probe islocated. This information may then be used to autoscale movements of thesurgical tools at the surgical site.

Keeping track of a position of the surgical tool relative to a tip ofthe endoscope using a focus encoder/actuator 8 (or other focus systemvehicles) may also be employed in the end effector controller describedherein. For example, the focusing mechanism state data may provideinformation regarding a Z axis dimension of the controller. X and Yposition information of the tool within a field of view of the endoscopecan be captured using image processing pattern recognition techniques.Alternative surgical tool location techniques may be described in U.S.Pat. No. 5,402,801, the full disclosure of which is also incorporatedherein by reference. In some embodiments, rather than tracking theposition of the surgical tool end effectors relative to the endoscopetip by assuming all of the positions of the robotic linkages relative toa common point on the base, the relative separation and/or positioningof the endoscope and tool (see FIG. 8) may at least in part bedetermined from the imaging system, for example, through patternrecognition and by determining the focal depth of the image capturedevice as focused on the surgical tool. Predicting tool position in thefinal displayed endoscope view analytically using robotic data (such asby summing the mechanical vectors of the linkages according to the jointstate data) may in at least some cases by unacceptably imprecise inpractice due to the combination of optical and mechanical tolerances anderror buildup. Optical recognition methods and system may reduce or evenbe immune to these tolerance or accuracy issues.

As noted above, many telesurgical systems may employ a plurality ofsurgical tools. Since the tool tips or end effectors are not alwayspositioned in the same plane perpendicular to longitudinal axis of theendoscope, one tool may be designated as the focal tool. The focal toolmay bear a master target on which the camera will automatically focus,and a camera may focus on that tool unless instructed otherwise byprocessor 5 and/or system operator O. The other tool(s) may carrysecondary targets, and if desired, may be designated by the operator asthe master tool on which the endoscope is to focus. While sometelerobotic systems may, for example, touch a tip of a probe or a toolto the tip of the endoscope to establish the relative positions of thesestructures, use of the data from image capture device 2 by, for example,focusing on the tool tip either manually or automatically, followed byusing position information from the image capture device and itsassociated manipulator may provide information regarding where the tooltip is located relative to the endoscope. The processor 5 may make useof this information to autoscale or adjust the movement scaling betweenthe input device and the end effector or tip of the tool.

The ability of processor 5 to determine absolute depths of focus mayalso enhance the ability of the surgeon to control the endoscopemovement. For example, endoscopes may be controlled by instructing theendoscope system to move verbally, followed by a verbal instruction tothe endoscope system to halt the movement. Similar instructions may begiven, for example, by actuating a joy stick or depressing a foot pedalfor the desired duration of movement and then releasing the joy stick orhalting the pressure on the foot pedal. Alternative input systems mayinstruct an endoscope to move an incremental distance for each verbal orother input instruction. The AESOP™ endoscope system commercially soldby Computer Motion of Goleta, Calif., and subsequently serviced byIntuitive Surgical of Sunnyvale, Calif., is an example of avoice-controlled endoscope system. These systems may, for example,benefit from techniques described herein so as to allow the systemoperator to (for example) verbally instruct the system to “focus oneinch,” “focus three inches,” or the like. Such instructions may providean absolute focus instruction, and similar absolute focus instructionsmay be input through other devices such as a keypad, input buttons, dialsettings, or the like.

Still further alternative embodiments may be provided. For example,stand alone voice activation may be provided so as to controlmagnification. A number of discrete positions may be available, orcontinuous zoom may be provided, allowing the system operator toinstruct the endoscope system to zoom in, zoom out, focus in, focus out,or the like, often through the use of a system in which the zoom and/orfocus is robotically driven as generally described above. In general, aplurality of specific positions (position 1, position 2, or the like) ormagnifications (wide, medium, narrow, or the like) may be provided withvoice activation optionally providing absolute (rather than relative)image capture device input. Along with altering the depth of focus inresponse to changes in magnification, information regarding the depth offocus may be used by the surgeon for a variety of uses in medicalapplications. The desired relative motion scaling between the inputdevice and end effector movements (or autoscale) may be determined basedon the focal depth for a variety of relationships. The relationship maybe linear, quadratic, or the like, and may be determined empirically,analytically, or from information provided by suppliers.

Referring now to FIG. 16, an exemplary method for adjusting focus and/ormotion scaling of a telerobotic or telesurgical system 502 begins withfocusing of the camera at an initial focus depth, as described above.The camera may be focused manually by the system operator or using anyof the wide variety of commercially available autofocus techniques. Afocal goal is established 506, such as maintaining the initial focuspoint at a fixed point in space, maintaining the focus at a movingrobotic tool structure, maintaining focus on a tissue, or the like. Suchfocals will often be embodied in a processor of the telerobotic systemusing software and/or hardware, and may be selected from a plurality offocal goals by the system operator.

In response to an input movement command from the system operator 508,the system processor calculates an appropriate movement of the imagecapture device and/or another robotic end effector 510. The effect ofthe robotic movement on focus and motion scaling is calculated 512, forexample, by determining a post-movement separation distance between theendoscope tip and the desired focus point.

Using the established focal goal 506 and the determined effect ofmovement on motion focus or scaling 512, the processor can calculate adesired change in focus and motion scaling. For example, where the goalis to maintain the focus at a fixed location in space, and the processorhas determined that the endoscope has moved so as to result in thecurrent focal state being focused at one inch short of the desired focuspoint, the processor can use the relationship between the focus stateand the desired focal distance graphically represented in FIG. 15 so asto calculate a change in the focal state. In other embodiments, when thechange in the position of the robotic arm results in the focus pointbeing half the distance to the endoscope camera as was present prior tothe move, with a linear relationship between separation distanceproducing a tool image as presented to the system operator which istwice as large after the endoscope movement, the system processor (forexample) change a motion scaling ratio between movement at an inputdevice and movement of the tool from 2:1 to 1:1. A variety ofalternative changes in desired focusing and/or motion scaling 514 maysimilarly be calculated.

Once the desired change in focus and/or motion scaling has beencalculated, the system processor sends signals to the focus or opticalscaling mechanism of the image capture device 516 so as to effect thechange. Changes to the focus or optical scaling of the image capturedevice may be effected by any of a wide variety of actuators, andconfirmation of the change may be transmitted to the processor using anencoder, potentiometer, or any of a wide variety of sensor or signaltransmitting structures.

While exemplary embodiments have been described in some detail forclarity of understanding and by way of example, a variety ofmodifications, changes, and adaptations will be obvious to those ofskill in the art. Hence, the scope of the present invention is limitedsolely by the appended claims.

What is claimed is:
 1. A robotic system comprising: an image capture device; an actuator configured to adjust a focus setting of the image capture device; a robotic linkage movably supporting the image capture device; at least one sensor for generating a sensor signal indicating movement of the robotic linkage; a processor configured to determine, based on the sensor signal, a change in distance between the image capture device and an initial focus point of the image capture device, the processor further configured to adjust the focus setting of the image capture device according to a predetermined relationship between the focus setting of the image capture device and the change in distance, the processor further configured to adjust the focus setting by controlling said actuator.
 2. The system of claim 1, the at least one sensor comprising an encoder of the robotic linkage.
 3. The system of claim 1, the at least one sensor comprising a potentiometer of the robotic linkage.
 4. The system of claim 1, the actuator configured to mechanically adjust the focus to a given focus setting, the processor further configured to adjust the focus setting by driving said actuator.
 5. The system of claim 1, the predetermined relationship being derived from parametric empirical measurements of the image capture device.
 6. The system of claim 1, further comprising a master input device, the robotic linkage configured to move said image capture device at least in part in response to movement of the master input device.
 7. The system of claim 1, the processor being configured to adjust the variable focus setting in response to an input from an operator of the system.
 8. The system of claim 7, the input comprising pressing a button on the master input device.
 9. The system of claim 1, wherein the predetermined relationship comprises a non-linear relationship between the focus setting and the change in distance.
 10. The system of claim 1, further comprising a plurality of image capture devices each having an adjustable focus setting, each image capture device removably coupleable to the linkage, the at least one sensor further comprising sensors for detecting movement of each image capture device, wherein the processor is further configured to adjust the focus of each image capture device in response to signals sent from the at least one sensor to the processor.
 11. The system of claim 1, wherein the image capture device has a first magnification setting and a second magnification setting, and wherein the processor is further configured to adjust the focus setting in response to a change in the magnification settings so as to maintain a focus point of the image capture device.
 12. The system of claim 1, further comprising a display coupled to the image capture device, wherein a display scale of an object shown on the display varies with a separation distance between a focal point of the image capture device and the image capture device.
 13. The system of claim 12, further comprising: an input device for inputting a master/slave movement command; the processor further configured to determine the movement corresponding to the movement command per a motion scale factor, the processor further configured to alter the motion scale factor in response to relative movement between the initial focus point and the image capture device so as to compensate for changes in the display scale.
 14. The system of claim 13, further comprising: an input device coupled to the processor for inputting a movement command; a robotic manipulator coupled to the processor, the processor further configured to determine a movement of the robotic manipulator corresponding to the movement command according to a motion scale factor, the processor further configured to alter the motion scale factor in response to the movement of the image capture device so as to compensate for changes in the display scale.
 15. The system of claim 1, the image capture device further having an adjustable magnification setting, the processor further configured to adjust the focus setting according to a predetermined relationship between the focus setting and a change in the magnification setting.
 16. The system of claim 15, the image capture device comprising a selectable magnification optical train having a plurality of alternative magnification lenses.
 17. The system of claim 15, the image capture device comprising a selectable magnification optical train having an electronic pixel variation system. 